Configuring surgical system with surgical procedures atlas

ABSTRACT

A surgical method is provided for use with a teleoperated surgical system (surgical system), the method comprising: recording surgical instrument kinematic information indicative of surgical instrument motion produced within the surgical system during the occurrence of the surgical procedure; determining respective kinematic signatures associated with respective surgical instrument motions; producing an information structure in a computer readable storage device that associates respective kinematic signatures with respective control signals; comparing, during a performance of the surgical procedure surgical instrument kinematic information during the performance with at least one respective kinematic signature; launching, during a performance of the surgical procedure an associated respective control signal in response to a match between surgical instrument kinematics during the performance and a respective kinematic signature.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims the benefit of priorityunder 35 U.S.C. § 120 to U.S. patent application Ser. No. 17/354,567,filed on Jun. 22, 2021, which is a continuation of and claims thebenefit of priority under 35 U.S.C. § 120 to U.S. patent applicationSer. No. 15/735,164, filed on Dec. 9, 2017, which is a U.S. NationalStage Filing under 35 U.S.C. 371 from International Application No.PCT/US2016/036733, filed on Jun. 9, 2016, and published as WO2016/201123 A1 on Dec. 15, 2016, which claims the benefit of priorityunder 35 U.S.C. § 119(e) to U.S. Provisional Patent Application No.62/173,077, filed on Jun. 9, 2015, each of which is hereby incorporatedby reference herein in its entirety.

BACKGROUND 1. Field of Invention

Inventive aspects are associated with medical devices used duringsurgery. More specifically, aspects are associated with controlling asurgical instrument in a robot-assisted surgical system based uponkinematic information and anatomic al tissue image information fromprior surgical procedures.

2. Art.

Surgeons typically undertake extensive study before performing asurgical procedure. Traditionally, surgeons were limited to the study ofgeneric anatomical models, such as photographs or drawings. Morerecently, various pre-operative diagnostic procedures (e.g., x-ray, CT,MRI, etc.) have made patient-specific anatomical information available.

In some cases, it is desirable to make additional, relevant anatomic andsurgical procedure information available to a surgeon. In one aspect, itis desirable to provide a surgeon planning an operation on a particularpatient with a surgical site video recording of an earlier surgicalprocedure performed on the particular patient. In another aspect, it isdesirable to provide a surgeon with one or more surgical videorecordings of surgical procedures on other patients that are similar tothe surgical procedure planned for a particular patient. In one aspect,it is desirable to provide such information to a surgeon prior to thesurgeon undertaking a particular surgical procedure. And in anotheraspect, it may be desirable to provide this information to a surgeonintraoperatively.

In one aspect, it is desirable to configure a video database thatincludes intraoperative surgical site video recordings of variousprocedures undergone by various patients. In one aspect, it is desirableto configure a medical device capable of video recording to furtherinclude an input that enables a surgeon using the medical device tohighlight and annotate the video recording in real time as it is beingrecorded. In one aspect, it is desirable to configure a computer-basedpattern matching algorithm to search through the individual records ofthe video database, identify relevant video records, and provide asurgeon with this relevant information for a particular surgicalprocedure.

SUMMARY

The following summary introduces certain aspects of the inventivesubject matter in order to provide a basic understanding. This summaryis not an extensive overview of the inventive subject matter, and it isnot intended to identify key or critical elements or to delineate thescope of the inventive subject matter. Although this summary containsinformation that is relevant to various aspects and embodiments of theinventive subject matter, its sole purpose is to present some aspectsand embodiments in a general form as a prelude to the more detaileddescription below.

In one aspect, a method is provided for use with a teleoperated surgicalsystem. Surgical instrument kinematic information that is indicative ofsurgical instrument motion is recorded for multiplicity of occurrencesof a surgical procedure. Kinematic signatures are determined based uponthe recorded kinematic information that are representative of surgicalinstrument motions. An information structure is produced in computerreadable storage device that associates respective kinematic signatureswith respective electronic control signals for a surgical system. Duringperformance of a surgical procedure using a surgical system, surgicalinstrument kinematic information produced by the system during theprocedure is compared with at least one kinematic signature. Anelectronic control signal associated with the at least one kinematicsignature is launched within the surgical system in response to a matchbetween kinematic information produced during the surgical procedure anda respective kinematic signature. More particularly, some systembehavior is triggered based upon kinematic analysis.

In another aspect, a method is provided for use with a teleoperatedsurgical system. Motion picture images of a surgical scene that areproduced during robot-assisted surgical procedure is recorded formultiplicity of occurrences of a surgical procedure. Surgical imagesignatures are determined based upon the recorded motion picture images.An information structure is produced in computer readable storage devicethat associates respective surgical image signatures with respectiveelectronic control signals for a surgical system. During performance ofa surgical procedure using a surgical system, motion picture imagesproduced during the procedure are compared with at least one surgicalimage signature. An electronic control signal associated with the atleast one surgical image signature is launched within the surgicalsystem in response to a match between surgical images produced duringthe surgical procedure and a respective surgical image signature. Moreparticularly, some system behavior is triggered based upon videoanalysis.

In another aspect, a training method is provided for use with ateleoperated surgical system. Motion picture images are recorded thatshow anatomical tissue within a surgical scene displayed within a viewerof a surgical system during a surgical procedure. Surgical systemcontrol haptics information, which is imparted to a surgical instrumentcontrol in response to a force imparted to a surgical instrument duringcontact with the displayed anatomical tissue, is recorded. The recordedmotion picture images are replayed within a surgical system viewerduring a simulation of the surgical procedure. The recorded surgicalinstrument control haptics are imparted to a surgical instrument controlduring the replaying of the recorded motion picture images during thesimulation of the surgical procedure. In some embodiments, surgicalcontrol haptics are replayed through vibro-tactile stimulation ofcontrol inputs.

In another aspect, a training method is provided for use with ateleoperated surgical system. Diagnosis data information instanceinstances are recorded for each of many occurrences of a surgicalprocedure within a robot-assisted surgical system. Each diagnosis datainformation instance includes respective motion picture images ofanatomical tissue within a surgical scene displayed within a viewer of asurgical system during a surgical procedure. Each diagnosis datainformation instance also includes surgical instrument control hapticsimparted to a surgical instrument control in response to a forceimparted to a surgical instrument during contact with the displayedanatomy during the surgical procedure. An information structure isproduced in a computer readable storage device that associatesrespective diagnosis data information instances with respectivediagnoses. A respective diagnosis data information instance is selected.Recorded motion picture images from the selected respective recordeddiagnosis data information instance are replayed within a surgicalsystem viewer during a simulation of the surgical procedure. Recordedsurgical instrument control haptics information from the selectedrespective recorded diagnosis data information instance is imparted to asurgical instrument control during the replaying of the recorded motionpicture images during the simulation of the surgical procedure.

In another aspect, a teleoperated surgical system includes aninformation structure in a computer readable storage device thatassociates surgical image signatures with control signals. A processoris configured to compare surgical images produced within a surgicalscene during a surgical procedure with at least one surgical imagesignature. The processor is configured to launch a control signal inresponse to a match between the surgical images and the at least onesurgical image signature. An instrument is configured to adjust itsmotion in response to the control signal.

In another aspect, a teleoperated surgical system includes aninformation structure in a computer readable storage device thatassociates respective surgical image signatures with respective controlsignals. A processor is configured to compare surgical images within asurgical scene during a surgical procedure with at least one surgicalimage signature. The processor is configured to launch a control signalin response to a match between surgical images during the surgicalprocedure and the at least one surgical image signature. An instrumentis configured to adjust its motion in response to the control signal.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a plan view of a minimally invasive teleoperated surgicalsystem.

FIG. 2 is a perspective view of a surgeon's console.

FIG. 3 is a perspective view of an electronics cart.

FIG. 4 is a diagrammatic illustration of a teleoperated surgical system.

FIG. 5A is an illustrative diagram of the teleoperated surgical system.

FIG. 5B is a perspective view of a patient-side cart of the surgicalsystem.

FIG. 5C is an illustrative view of a surgical scene.

FIG. 6 is an elevation view of a surgical instrument.

FIG. 7 is a perspective view of an instrument manipulator.

FIG. 8 is a diagrammatic illustration of a surgical planning tool.

FIG. 9 is a flow diagram of a method of using a surgical planning tool.

FIG. 10 is an illustrative drawing representing storage atlas in acomputer readable storage device in accordance with some embodiments.

FIG. 11 is an illustrative drawing representing an example instance of afirst data information structure included within the atlas, whichincludes information about an individual surgical procedure inaccordance with some embodiments.

FIG. 12 is an illustrative drawing representing an example instance ofthe second data information structure included within the atlas, whichassociates recorded motion picture image segments from an individualsurgical procedure, corresponding surgical instrument kinematicsinformation segments, corresponding surgical system actuation states,and corresponding annotations, in accordance with some embodiments.

FIGS. 13A-13C are illustrative drawings showing an example surgicalinstrument and an actuator assembly in which the surgical instrument isshown in three different example instrument states in accordance withsome embodiments.

FIG. 14 is an illustrative drawing representing an example instance ofcontrol signal rules information structure, which includes rules toassociate kinematics, images and system actuation states with controlsignals in accordance with some embodiments.

FIG. 15 is an illustrative flow diagram representing a process toproduce a control signal based at least in part upon anatomical imageinformation, instrument kinematics and system state, in accordance withsome embodiments.

FIG. 16 is an illustrative drawing representing an example instance ofthe third data information structure included within the atlas, whichassociates recorded motion picture image segments with hapticsinformation segments, in accordance with some embodiments.

FIG. 17 is an illustrative flow diagram representing a process toconfigure a teleoperated robot-assisted surgical system to playback asurgical experience, in accordance with some embodiments.

FIG. 18 is an illustrative drawing representing an example instance ofthe diagnosis rules information structure included within the atlas,which includes rules to associate image signature information and hapticfeedback information with diagnoses, in accordance with someembodiments.

FIG. 19 is an illustrative flow diagram representing a process toproduce a diagnosis based at least in part upon anatomical imageinformation and haptic feedback information, in accordance with someembodiments.

DETAILED DESCRIPTION

This description and the accompanying drawings that illustrate inventiveaspects, embodiments, implementations, or applications should not betaken as limiting—the claims define the protected invention. Variousmechanical, compositional, structural, electrical, and operationalchanges may be made without departing from the scope of this descriptionand the claims. In some instances, well-known circuits, structures, ortechniques have not been shown or described in detail in order not toobscure the invention. Like numbers in two or more figures represent thesame or similar elements.

Elements described in detail with reference to one embodiment,implementation, or application may, whenever practical, be included inother embodiments, implementations, or applications in which they arenot specifically shown or described. For example, if an element isdescribed in detail with reference to one embodiment and is notdescribed with reference to a second embodiment, the element maynevertheless be claimed as included in the second embodiment. Thus, toavoid unnecessary repetition in the following description, one or moreelements shown and described in association with one embodiment,implementation, or application may be incorporated into otherembodiments, implementations, or aspects unless specifically describedotherwise, unless the one or more elements would make an embodiment orimplementation non-functional, or unless two or more of the elementsprovide conflicting functions.

Aspects of the invention are described primarily in terms of animplementation using a da Vinci® Surgical System (specifically, a ModelIS4000, marketed as the da Vinci® Xi™ HD™ Surgical System),commercialized by Intuitive Surgical, Inc. of Sunnyvale, California.Knowledgeable persons will understand, however, that inventive aspectsdisclosed herein may be embodied and implemented in various ways,including robotic and, if applicable, non-robotic embodiments andimplementations. Implementations on da Vinci® Surgical Systems (e.g.,the Model IS4000 da Vinci® Xi™ Surgical System, the Model IS3000 daVinci Si® Surgical System) are merely exemplary and are not to beconsidered as limiting the scope of the inventive aspects disclosedherein.

In accordance with various aspects, the present disclosure describes asurgical planning tool that includes a medical device configured tovideo record the performance of surgical procedures. The videorecordings can be embedded with various metadata, e.g., highlights madeby a medical person. Additionally, the video recordings can be taggedwith various metadata, e.g., text annotations describing certain subjectmatter of the video, the identity of the patient to whom the videorecording corresponds, biographical or medical information about thepatient, and the like. In one aspect, tagged metadata is embedded in thevideo recordings.

In accordance with further aspects, information patterns are identifiedwithin motion picture images and surgical instrument kinematicinformation collected from numerous teleoperated surgical procedures.Motion picture information can indicate anatomical tissue geometry andcoloration, for example. Kinematic information can indicate surgicalinstrument motion characteristics such as direction of instrumentmotion, speed and acceleration of instrument motion, and sequences ofinstrument motion, for example. The information patterns can beidentified based upon the recorded motion picture and kinematicinformation can be used as a basis to manage or regulate controlsurgical instrument during surgery. The information patterns can be usedas a basis to provide intra-surgical guidance to a surgeon.

In accordance with further aspects, motion picture images in concertwith haptic feedback can be used as bases for surgical training. Forexample, a surgeon can re-experience a prior surgical procedureperformed by that surgeon through a surgical simulation that replaysmotion picture images and corresponding haptic feedback produced duringthe prior surgery by that same surgeon. Alternatively, for example, asurgeon can experience a previous surgical procedure performed by adifferent surgeon through a surgical simulation that replays motionpicture images and corresponding haptic feedback produced during thatprevious surgery by another. Thus, a surgeon can use a surgicalsimulation as an opportunity to refine surgical skills through asimulated practice surgery that replays an actual surgical experience bythat surgeon or another surgeon of relationship between visual cues andhaptic cues.

In accordance with still further aspects, proposed intra-surgicaldiagnoses are developed based upon information patterns identifiedwithin motion picture images and surgical instrument kinematicinformation collected from numerous teleoperated surgical procedures. Askilled surgeon often can evaluate tissue disease state and tissuetrauma state based at least in part upon tissue geometry and coloration.Recorded motion pictures provide information as to tissue geometry andtissue coloration of anatomical tissue within a surgical scene within asurgical system. Moreover, a skilled surgeon can evaluate tissue diseasestate and tissue trauma state based at least in part upon palpation ofthe tissue. In a teleoperated surgical system, palpation of tissue canbe achieved through touch upon a tissue structure using asurgeon-operated instrument control that provides haptic feedback to asurgeon operating the control that is indicative of reactive forceimparted to the instrument in response to the instrument touch upon thetissue structure. Expert surgeon evaluation of the collected motionpicture images and surgical instrument kinematic information is used toidentify different patterns of images and kinematics indicative ofdifferent intra-surgical diagnoses. The video recordings and informationstructures that associate motion picture images with surgical instrumentkinematics information can be archived on an electronic medical recorddatabase implemented locally or remotely (e.g., on a remote computersystem on a LAN or WAN, or on a cloud data storage service). Similarly,in some embodiments, information structures that associate motionpicture images with control haptics feedback information andcorresponding diagnosis recommendations can be archived on an electronicmedical record database implemented locally or remotely for use insurgeon training, for example. The video recordings and informationstructures can be made available to interested health care providers. Insome embodiments, stored information structures can be made availablefor use with a teleoperated robot assisted surgical system to generatecontrol signal information to provide to a surgical system to produceintra-surgery surgical guidance to a surgeon and to providerobot-assisted surgical control of instruments during a surgicalprocedure. In some embodiments, stored information structures can bemade available for use with a surgical simulation system to replaysurgical scenes and corresponding haptic feedback for use in surgeontraining in mechanics of operating a surgical system. In someembodiments, stored information structures can be made available for usewith a surgical simulation system to replay surgical scenes andcorresponding haptic feedback for use in surgeon training in diagnosisof tissue structure disease state and tissue trauma state whileperforming a surgery using the surgical system.

Health care providers can search the medical device database based uponone or more of surgical procedures to be performed, tissue structurecharacteristics, and surgical instrument kinematics for videos andinformation structure relationships of interest using the metadata tagsdescribed above, for example. Additionally, in one aspect, the surgicalplanning tool includes a computer-based pattern matching and analysisalgorithm. In one aspect, the pattern-matching algorithm culls throughthe videos stored on the electronic medical record database to identifycorrelations between visual characteristics in the video recordings andassociated metadata tags made by medical persons. The surgical planningtool can apply these correlations to newly encountered anatomy, andthereby assist medical persons performing a procedure in makingdeterminations about patient anatomy, preferred surgical approaches,disease states, potential complications, etc.

In another aspect, a pattern matching algorithm culls through recordedmotion picture image information and, optionally, kinematic informationto identify correlations between anatomical tissue features such asgeometry and instrument motion, for example. Such patterns can beuseful, for example, to identify kinds of anatomical features associatedwith kinds of instrument motion. Such patterns also can be useful, forexample, to identify kinds of anatomical features that are notassociated with kinds of instrument motion. Such pattern information canbe used as a basis to produce surgical guidance to present to a surgeonduring a surgery, for example. Such pattern information can be used as abasis to deter or to impart surgical certain surgical instrument motionduring a surgery, for example.

In another aspect, a pattern matching algorithm culls through recordedmotion picture image information and haptic feedback information toidentify correlations between anatomical tissue features such asgeometry and reactive force imparted by the tissue structure in responseto touch by a surgical instrument, for example. Such patterns can beuseful, for example, to identify correlations between visible anatomicaltissue structures and haptic feedback imparted by the tissue structurein response to palpation by a robot-assisted instrument. In someembodiments, correlated motion picture image patterns and hapticfeedback information are associated with expert surgeon diagnosisevaluations for use in surgeon training.

Minimally Invasive Teleoperated Surgical System

Teleoperation refers to operation of a machine at a distance. In aminimally invasive teleoperation medical system, a surgeon may use anendoscope that includes a camera to view a surgical site within apatient's body. In some embodiments, stereoscopic images can becaptured, which allow the perception of depth during a surgicalprocedure.

Referring now to the drawings, in which like reference numeralsrepresent like parts throughout the several views, FIG. 1 is a plan viewof a minimally invasive teleoperated surgical system 10, typically usedfor performing a minimally invasive diagnostic or surgical procedure ona patient 12 who is lying on an operating table 14. The system includesa surgeon's console 16 for use by a surgeon 18 during the procedure. Oneor more assistants 20 may also participate in the procedure. Theminimally invasive teleoperated surgical system 10 further includes apatient-side cart(s) 22 and an electronics cart 24. The patient-sidecart 22 can manipulate at least one surgical instrument 26 through aminimally invasive incision in the body of the patient 12 while thesurgeon 18 views the surgical site through the surgeon's console 16. Animage of the surgical site can be obtained by an endoscope 28, such as astereoscopic endoscope, which can be manipulated by the patient-sidecart 22 to orient the endoscope 28. Computer processors located on theelectronics cart 24 can be used to process the images of the surgicalsite for subsequent display to the surgeon 18 through the surgeon'sconsole 16. Note that while discrete system components (i.e., patientside cart 22, electronics cart 24, and surgeon's console 16) aredepicted and described for exemplary purposes, in various embodimentsthe elements included therein can be combined and/or separated. Forexample, in some embodiments, the computer processors of electronicscart 24 can be incorporated into surgeon's console 16 and/or patientside cart 22. The number of surgical instruments 26 used at one timewill generally depend on the diagnostic or surgical procedure and thespace constraints within the operative site among other factors. If itis necessary to change one or more of the surgical instruments 26 beingused during a procedure, an assistant 20 can remove the surgicalinstrument 26 from the patient-side cart 22, and replace it with anothersurgical instrument 26 from a tray 30 in the operating room.

FIG. 2 is a perspective view of the surgeon's console 16. The surgeon'sconsole 16 includes a viewer 31 that includes a left eye display 32 anda right eye display 34 for presenting the surgeon 18 with a coordinatedstereoscopic view of the surgical site that enables depth perception.The console 16 further includes one or more control inputs 36. One ormore surgical instruments installed for use on the patient-side cart 22(shown in FIG. 1 ) move in response to surgeon 18's manipulation of theone or more control inputs 36. The control inputs 36 can provide thesame mechanical degrees of freedom as their associated surgicalinstruments 26 (shown in FIG. 1 ) to provide the surgeon 18 withtelepresence, or the perception that the control inputs 36 are integralwith the instruments 26 so that the surgeon has a strong sense ofdirectly controlling the instruments 26. To this end, position, force,and tactile feedback sensors (not shown) may be employed to transmitposition, force, and tactile sensations from the surgical instruments 26back to the surgeon's hands through the control inputs 36, subject tocommunication delay constraints. Note that while a physical console 16with a fixed viewer 31 and mechanically coupled control inputs 36 isdepicted and described for exemplary purposes, in various otherembodiments, “ungrounded” control inputs and/or display structures canbe used. For example, in some embodiments, viewer 31 can be ahead-mounted display and/or control inputs 36 can be mechanicallyindependent of any base structure (e.g., wired, wireless, orgesture-based, such as Kinect from Microsoft).

The surgeon's console 16 is usually located in the same room as thepatient so that the surgeon can directly monitor the procedure, bephysically present if necessary, and speak to a patient-side assistantdirectly rather than over the telephone or other communication medium.But, the surgeon can be located in a different room, a completelydifferent building, or other remote location from the patient allowingfor remote surgical procedures.

FIG. 3 is a perspective view of the electronics cart 24. The electronicscart 24 can be coupled with the endoscope 28 and includes a computerprocessor to process captured images for subsequent display, such as toa surgeon on the surgeon's console, or on another suitable displaylocated locally and/or remotely. For example, if a stereoscopicendoscope is used, a computer processor on electronics cart 24 canprocess the captured images to present the surgeon with coordinatedstereo images of the surgical site. Such coordination can includealignment between the opposing images and can include adjusting thestereo working distance of the stereoscopic endoscope. As anotherexample, image processing can include the use of previously determinedcamera calibration parameters to compensate for imaging errors of theimage capture device, such as optical aberrations. Optionally, equipmentin electronics cart may be integrated into the surgeon's console or thepatient-side cart, or it may be distributed in various other locationsin the operating room.

FIG. 4 diagrammatically illustrates a teleoperated surgical system 50(such as the minimally invasive teleoperated surgical system 10 of FIG.1 ). A surgeon's console 52 (such as surgeon's console 16 in FIG. 1 )can be used by a surgeon to control a patient-side cart 54 (such aspatent-side cart 22 in FIG. 1 ) during a minimally invasive procedure.The patient-side cart 54 can use an imaging device, such as astereoscopic endoscope, to capture images of a surgical site and outputthe captured images to a computer processor located on an electronicscart 56 (such as the electronics cart 24 in FIG. 1 ). The computerprocessor typically includes one or more data processing boards purposedfor executing computer readable code stored in a non-volatile memorydevice of the computer processor. In one aspect, the computer processorcan process the captured images in a variety of ways prior to anysubsequent display. For example, the computer processor can overlay thecaptured images with a virtual control interface prior to displaying thecombined images to the surgeon via the surgeon's console 52.

Additionally or in the alternative, the captured images can undergoimage processing by a computer processor located outside of electronicscart 56. In one aspect, teleoperated surgical system 50 includes anoptional computer processor 58 (as indicated by dashed line) similar tothe computer processor located on electronics cart 56, and patient-sidecart 54 outputs the captured images to computer processor 58 for imageprocessing prior to display on the surgeon's console 52. In anotheraspect, captured images first undergo image processing by the computerprocessor on electronics cart 56 and then undergo additional imageprocessing by computer processor 58 prior to display on the surgeon'sconsole 52. Teleoperated surgical system 50 can include an optionaldisplay 60, as indicated by dashed line. Display 60 is coupled with thecomputer processor located on the electronics cart 56 and with computerprocessor 58, and captured images processed by these computer processorscan be displayed on display 60 in addition to being displayed on adisplay of the surgeon's console 52.

FIG. 5A is an illustrative simplified block diagram showing arrangementof components of the teleoperation surgery system 10 to perform surgicalprocedures using one or more mechanical support arms 510 in accordancewith some embodiments. Aspects of system 10 includes robot-assisted andautonomously operating features. These mechanical support arms 510 oftensupport a surgical instrument. For instance, a mechanical surgical arm(e.g., the center mechanical surgical arm 510C) may be used to supportan endoscope with a stereo or three-dimensional surgical image capturedevice 101C. The mechanical surgical arm 510C may include a sterileadapter, or a clamp, clip, screw, slot/groove, or other fastenermechanism to mechanically secure an endoscope that includes the imagecapture device 101C to the mechanical arm.

A user or operator O (generally a surgeon) performs a surgical procedureon patient P by manipulating control input devices 36, such as handgrips and foot pedals at a master control console 16. The operator canview video frames of images of a surgical site inside a patient's bodythrough a stereo display viewer 31. A computer processor 58 of theconsole 16 directs movement of teleoperationally controlled endoscopicsurgical instruments 101A-101C via control lines 159, effecting movementof the instruments using a patient-side system 24 (also referred to as apatient-side cart).

The patient-side system 24 includes one or more mechanical support arms510. Typically, the patient-side system 24 includes at least threemechanical surgical arms 510A-510C (generally referred to as mechanicalsurgical support arms 510) supported by corresponding positioning set-uparms 156. The central mechanical surgical arm 510C may support anendoscopic camera 101C suitable for capture of images within a field ofview of the camera. The mechanical surgical support arms 510A and 510Bto the left and right of center may support instruments 101A and 101B,respectively, which may manipulate tissue.

FIG. 5B is a perspective view of a patient-side cart 500 of a minimallyinvasive teleoperated surgical system 10, in accordance withembodiments. The patient-side cart 500 includes one or more support armassemblies 510. A surgical instrument manipulator 512 is mounted at theend of each support arm assembly 510. Additionally, each support armassembly 510 can optionally include one or more setup joints (e.g.,unpowered and/or lockable) that are used to position the attachedsurgical instrument manipulator 512 with reference to the patient forsurgery. As depicted, the patient-side cart 500 rests on the floor. Inother embodiments, operative portions of the patient-side cart can bemounted to a wall, to the ceiling, to the operating table 526 that alsosupports the patient's body 522, or to other operating room equipment.Further, while the patient-side cart 500 is shown as including foursurgical instrument manipulators 512, more or fewer surgical instrumentmanipulators 512 may be used.

A functional teleoperated surgical system will generally include avision system portion that enables a user of the teleoperated surgicalsystem to view the surgical site from outside the patient's body 522.The vision system typically includes a camera instrument 528 forcapturing video images and one or more video displays for displaying thecaptured video images. In some surgical system configurations, thecamera instrument 528 includes optics that transfer the images from adistal end of the camera instrument 528 to one or more imaging sensors(e.g., CCD or CMOS sensors) outside of the patient's body 522.Alternatively, the imaging sensor(s) can be positioned at the distal endof the camera instrument 528, and the signals produced by the sensor(s)can be transmitted along a lead or wirelessly for processing and displayon the one or more video displays. One example of a video display is thestereoscopic display on the surgeon's console in surgical systemscommercialized by Intuitive Surgical, Inc., Sunnyvale, California.

Referring to FIGS. 5A-5B, mounted to each surgical instrumentmanipulator 512 is a surgical instrument 520 that operates at a surgicalsite within the patient's body 522. Each surgical instrument manipulator512 can be provided in a variety of forms that allow the associatedsurgical instrument to move with one or more mechanical degrees offreedom (e.g., all six Cartesian degrees of freedom, five or fewerCartesian degrees of freedom, etc.). Typically, mechanical or controlconstraints restrict each manipulator 512 to move its associatedsurgical instrument around a center of motion on the instrument thatstays stationary with reference to the patient, and this center ofmotion is typically located at the position where the instrument entersthe body.

In one aspect, surgical instruments 520 are controlled throughcomputer-assisted teleoperation. A functional minimally invasiveteleoperated surgical system includes a control input that receivesinputs from a user of the teleoperated surgical system (e.g., a surgeonor other medical person). The control input is in communication with oneor more computer-controlled teleoperated actuators, such as one or moremotors to which surgical instrument 520 is coupled. In this manner, thesurgical instrument 520 moves in response to a medical person'smovements of the control input. In one aspect, one or more controlinputs are included in a surgeon's console such as surgeon's console 16shown at FIG. 2 . A surgeon can manipulate control input devices 36 ofsurgeon's console 16 to operate teleoperated actuators of patient-sidecart 500. The forces generated by the teleoperated actuators aretransferred via drivetrain mechanisms, which transmit the forces fromthe teleoperated actuators to the surgical instrument 520.

Referring to FIGS. 5A-5B, in one aspect, a surgical instrument 520 and acannula 524 are removably coupled to manipulator 512, with the surgicalinstrument 520 inserted through the cannula 524. One or moreteleoperated actuators of the manipulator 512 move the surgicalinstrument 512 as a whole. The manipulator 512 further includes aninstrument carriage 530. The surgical instrument 520 is detachablyconnected to the instrument carriage 530. In one aspect, the instrumentcarriage 530 houses one or more teleoperated actuators inside thatprovide a number of controller motions that the surgical instrument 520translates into a variety of movements of an end effector on thesurgical instrument 520. Thus the teleoperated actuators in theinstrument carriage 530 move only one or more components of the surgicalinstrument 520 rather than the instrument as a whole. Inputs to controleither the instrument as a whole or the instrument's components are suchthat the input provided by a surgeon or other medical person to thecontrol input (a “master” command) is translated into a correspondingaction by the surgical instrument (a “slave” response).

In accordance with some embodiments, the surgical system 10 can havemultiple system actuation states including docked, following, instrumenttypes and head-in. During a docked system state, one or more manipulator512 have been coupled to cannula 524. During a following system state,the surgical instrument (“slave”) is tracking the control input(“master” command). During an instrument-types system state, the systemthe system has installed in it a set of instruments suitable forperformance of a particular surgical procedure or suitable forperformance of a particular surgical activity during a surgicalprocedure. During a head-in system state, the system is waiting for thesurgeon to indicate he/she has taken hold of the “master” control inputdevice.

In an alternate embodiment, instrument carriage 530 does not houseteleoperated actuators. Teleoperated actuators that enable the varietyof movements of the end effector of the surgical instrument 520 arehoused in a location remote from the instrument carriage 530, e.g.,elsewhere on patient-side cart 500. A cable-based force transmissionmechanism or the like is used to transfer the motions of each of theremotely located teleoperated actuators to a correspondinginstrument-interfacing actuator output located on instrument carriage530. In some embodiments, the surgical instrument 520 is mechanicallycoupled to a first actuator, which controls a first motion of thesurgical instrument such as longitudinal (z-axis) rotation. The surgicalinstrument 520 is mechanically coupled to a second actuator, whichcontrols second motion of the surgical instrument such astwo-dimensional (x, y) motion. The surgical instrument 520 ismechanically coupled to a third actuator, which controls third motion ofthe surgical instrument such as opening and closing or a jaws endeffector.

FIG. 5C is an illustrative view representing a surgical scene 550 andalso showing an endoscope 101C mounting a camera 528 used to record thescene in accordance with some embodiments. The scene 550 is disposedwithin a patient's body cavity. The scene 550 includes an examplehypothetical spherical anatomical structure 552 that includes geometriccontours 554. The scene 550 encompasses a surgical instrument 556. Acamera 528 mounted on an endoscope 101C captures the scene, which isdisplayed within the viewer 31 and which is recorded for playback later.

FIG. 6 is a side view of a surgical instrument 520, which includes adistal portion 650 and a proximal control mechanism 640 coupled by anelongate tube 610 having an elongate tube centerline axis 611. Thesurgical instrument 520 is configured to be inserted into a patient'sbody and is used to carry out surgical or diagnostic procedures. Thedistal portion 650 of the surgical instrument 520 can provide any of avariety of end effectors 654, such as the forceps shown, a needledriver, a cautery device, a cutting tool, an imaging device (e.g., anendoscope or ultrasound probe), or the like. The surgical end effector654 can include a functional mechanical degree of freedom, such as jawsthat open or close, or a knife that translates along a path. In theembodiment shown, the end effector 654 is coupled to the elongate tube610 by a wrist 652 that allows the end effector to be oriented relativeto the elongate tube centerline axis 611. Surgical instrument 520 canalso contain stored (e.g., on a semiconductor memory associated with theinstrument) information, which may be permanent or may be updatable by asurgical system configured to operate the surgical instrument 520.Accordingly, the surgical system may provide for either one-way ortwo-way information communication between the surgical instrument 520and one or more components of the surgical system.

FIG. 7 is a perspective view of surgical instrument manipulator 512.Instrument manipulator 512 is shown with no surgical instrumentinstalled. Instrument manipulator 512 includes an instrument carriage530 to which a surgical instrument (e.g., surgical instrument 520) canbe detachably connected. Instrument carriage 530 houses a plurality ofteleoperated actuators. Each teleoperated actuator includes an actuatoroutput 705. When a surgical instrument is installed onto instrumentmanipulator 512, one or more instrument inputs (not shown) of aninstrument proximal control mechanism (e.g., proximal control mechanism640 at FIG. 6 ) are mechanically coupled with corresponding actuatoroutputs 705. In one aspect, this mechanical coupling is direct, withactuator outputs 705 directly contacting corresponding instrumentinputs. In another aspect, this mechanical coupling occurs through anintermediate interface, such as a component of a drape configured toprovide a sterile barrier between the instrument manipulator 512 anassociated surgical instrument.

In one aspect, movement of one or more instrument inputs bycorresponding teleoperated actuators results in a movement of a surgicalinstrument mechanical degree of freedom. For example, in one aspect, thesurgical instrument installed on instrument manipulator 512 is surgicalinstrument 520, shown at FIG. 6 . Referring to FIG. 6 , in one aspect,movement of one or more instrument inputs of proximal control mechanism640 by corresponding teleoperated actuators rotates elongate tube 610(and the attached wrist 652 and end effector 654) relative to theproximal control mechanism 640 about elongate tube centerline axis 611.In another aspect, movement of one or more instrument inputs bycorresponding teleoperated actuators results in a movement of wrist 652,orienting the end effector 654 relative to the elongate tube centerlineaxis 611. In another aspect, movement of one or more instrument inputsby corresponding teleoperated actuators results in a movement of one ormore moveable elements of the end effector 654 (e.g., a jaw member, aknife member, etc.). Accordingly, various mechanical degrees of freedomof a surgical instrument installed onto an instrument manipulator 512can be moved by operation of the teleoperated actuators of instrumentcarriage 530.

Annotating a Recorded Video

FIG. 8 shows a schematic diagram of an exemplary surgical planning tool800. In one aspect, surgical planning tool 800 includes a teleoperatedsurgical system 50 in data communication with an electronic medicaldevice record database 830. Teleoperated surgical system 50 shown hereis similar to teleoperated surgical system 50 shown at FIG. 4 . In oneaspect, electronic medical record database 830 includes the medicalrecords of patients that have undergone treatment at a particularhospital or at a plurality of hospitals. Database 830 can be implementedon a server located on-site at the hospital. The medical record entriescontained in the database 830 can be accessed from hospital computersthrough an intranet network. Alternatively, database 830 can beimplemented on a remote server located off-site from the hospital, e.g.,using one of a number of cloud data storage services. In this case,medical record entries of database 830 are stored on the cloud server,and can be accessed by a computer with internet access.

In one aspect, a surgical procedure is performed on a first patientusing teleoperated surgical system 50. An imaging device associated withteleoperated surgical system 50 captures images of the surgical site anddisplays the captured images as frames of a video on a display ofsurgeon's console 52. In one aspect, a medical person at surgeon'sconsole 52 highlights or annotates certain patient anatomy shown in thedisplayed video using an input device of surgeon's console 52. Anexample of such an input device is control input 36 shown at FIG. 2 ,which is coupled to a cursor that operates in conjunction with a graphicuser interface overlaid onto the displayed video. The graphic userinterface can include a QWERTY keyboard, a pointing device such as amouse and an interactive screen display, a touch-screen display, orother means for data or text entry or voice annotation/or speech to textconversion via a microphone and processor. Accordingly, the medicalperson can highlight certain tissue of interest in the displayed imageor enter a text annotation.

In one aspect, the surgical site video is additionally displayed on adisplay located on electronics cart 56. In one aspect, the display ofelectronics cart is a touch-screen user interface usable by a medicalperson to highlight and annotate certain portions of patient anatomyshown on an image that is displayed for viewing on the display on theelectronics cart. A user, by touching portions of patient anatomydisplayed on the touch-screen user interface, can highlight portions ofthe displayed image. Additionally, a graphic interface including aQWERTY keyboard can be overlaid on the displayed image. A user can usethe QWERTY keyboard to enter text annotations.

In one aspect, the surgical site video captured by the imaging deviceassociated with teleoperated surgical system 50 is recorded by theteleoperated surgical system 50, and stored on database 830, in additionto being displayed in real time or near real time to a user. Highlightsand/or annotations associated with the recorded video that were made bythe user can also be stored on database 830. In one aspect, thehighlights made by the user are embedded with the recorded video priorto its storage on database 830. At a later time, the recorded video canbe retrieved for viewing. In one aspect, a person viewing the recordedvideo can select whether the highlights are displayed or suppressed fromview. Similarly, annotations associated with the recorded video can alsobe stored on database 830. In one aspect, the annotations made by theuser are used to tag the recorded video, and can be used to provide as ameans of identifying the subject matter contained in the recorded video.For example, one annotation may describe conditions of a certain diseasestate. This annotation is used to tag the recorded video. At a latertime, a person desiring to view recorded procedures concerning thisdisease state can locate the video using a key word search.

Retrieval of Stored Video

In some cases, it is desirable for a medical person to be able to viewvideo recordings of past surgical procedures performed on a givenpatient. In one aspect, a patient who previously underwent a firstsurgical procedure to treat a medical condition subsequently requires asecond surgical procedure to treat recurrence of the same medicalcondition or to treat anatomy located nearby to the surgical site of thefirst surgical procedure. In one aspect, the surgical site events of thefirst surgical procedure were captured in a surgical site videorecording, and the video recording was archived in database 830 as partof the patient's electronic medical records. Prior to performing thesecond surgical procedure on the patient, a medical person can perform asearch of database 830 to locate the video recording of the patient'searlier surgical procedure.

In some cases, it is desirable for a medical person planning to performa surgical procedure on a patient to be able to view video recordings ofsimilar surgical procedures performed on persons having certaincharacteristics similar to the patient. In one aspect, surgical sitevideo recordings of surgical procedures can be tagged with metadatainformation such as the patient's age, gender, body mass index, geneticinformation, type of procedure the patient underwent, etc., before eachvideo recording is archived in database 830. In one aspect, the metadatainformation used to tag a video recording is automatically retrievedfrom a patient's then-existing medical records, and then used to tag thevideo recording before the video recording is archived in database 830.Accordingly, prior to performing a medical procedure on a patient, amedical person can search database 830 for video recordings of similarprocedures performed on patients sharing certain characteristics incommon with the patient. For example, if the medical person is planningto use teleoperated surgical system 50 to perform a prostatectomy on a65 year-old male patient with an elevated body mass index, the medicalperson can search database 830 for surgical site video recordings ofprostatectomies performed using teleoperated surgical system 50 on othermales of similar age and having similarly elevated body mass index.

In one aspect, a video recording of a surgical procedure is communicatedby database 830 to an optional personal computer 820 (as indicated bydashed line), and made available for viewing by a medical person whoplans to perform a surgical procedure. Additionally or in thealternative, the video recording of the earlier surgical procedure canbe communicated by database 830 to teleoperated surgical system 50, andmade available for viewing preoperatively or intraoperatively. In oneaspect, the video recording is displayed by teleoperated surgical system50 on a display located on surgeon's console 52. In another aspect, thevideo recording of the first surgical procedure is displayed on adisplay located on electronics cart 56.

Cloud-Based Video Database

In one aspect, database 830 is implemented on a remote server using acloud data storage service and is accessible by multiple health careproviders. Referring to FIG. 8 , as shown by dashed line, surgicalplanning tool 800 optionally includes teleoperated surgical system 850(as indicated by dashed line) and personal computer 840 (as indicated bydashed line). In one aspect, teleoperated surgical system 850 is similarto teleoperated surgical system 50 and personal computer 840 is similarto personal computer 820, except that teleoperated surgical system 50and personal computer 820 are located at a first health care providerand teleoperated surgical system 850 and personal computer 840 arelocated at a second location or even with a second health care provider.In one aspect, a first patient requires surgical treatment of a medicalcondition, and undergoes a surgical procedure using teleoperatedsurgical system 50 at the first health care provider. A video recordingof the surgical procedure is archived on database 830. At a later time,a second patient requires surgical treatment of the same medicalcondition, and plans to receive surgical treatment using teleoperatedsurgical system 850 at the second health care provider. Prior toperforming the surgical procedure on the second patient, a medicalperson accesses database 830 through a secure internet connection andsearches database 830 for surgical site video recordings of similarprocedures. In one aspect, the medical person treating the secondpatient is able to retrieve from database 830 the video recording offirst patient's surgical procedure, without acquiring knowledge of theidentity of the first patient. In this manner, the privacy of the firstpatient is maintained. In one aspect, the video recording of the firstpatient's surgical procedure includes highlights and/or annotations madeby the medical person who treated the first patient.

Computer Based Pattern Matching and Analysis

Surgical planning tool 800 can includes a pattern matching and analysisalgorithm implemented in the form of computer executable code. In oneaspect, the pattern matching and analysis algorithm is stored in anon-volatile memory device of surgical planning tool 800, and isconfigured to analyze the video recordings archived in database 830. Asdiscussed previously, each of the video recordings archived in database830 can be tagged and/or embedded with certain metadata information.This metadata information can include patient information such aspatient age, gender, and other information describing the patient'shealth or medical history. Additionally, as discussed previously, themetadata information can include highlights or annotations made by amedical person. In one aspect, these highlights and annotations areembedded with the video recording and archived together with the videoin database 830.

In one aspect, pattern matching and analysis algorithm includes an imageanalysis component that identifies patterns in shapes and colors thatare shared amongst multiple video recordings stored on database 830. Thepattern matching and analysis algorithm then reviews the tagged metadataassociated with this subset of video recordings to determine whether anywords or phrases are frequently associated with videos within thissubset. These analyses performed by pattern matching and analysisalgorithm can be used to assist medical persons in making determinationsabout patient anatomy, preferred surgical approaches, disease states,potential complications, etc.

A Method of Using a Surgical Planning Tool

FIG. 9 shows a method 900 of using a surgical planning tool. In oneaspect, the surgical planning tool is similar to surgical planning tool800 at FIG. 8 . At 910, a fact or characteristic describing a medicalpatient, e.g., a medical condition suffered by a patient, is received bya medical device. The medical device can receive this fact orcircumstance via a user interface located on a teleoperated surgicalsystem (e.g., teleoperated surgical system 10 at FIG. 1 or teleoperatedsurgical system 50 at FIG. 4 ), or alternatively, through a personalcomputer similar to personal computer 820 at FIG. 8 . At 920, themedical device uses the fact or characteristic received at 910 toretrieve at least one relevant video recording of a surgical procedurefrom a medical device database. At 930, the medical device uses thevideo recordings to determine surgical planning information. In oneaspect, the surgical planning information includes the types ofinstruments used in the recorded procedure. At 940, the medical devicedisplays to a user the surgical planning information determined at 930.

FIG. 10 is an illustrative drawing representing storage atlas 1002 in acomputer readable storage device 1004 in accordance with someembodiments. The storage atlas 1002 includes first data informationstructures 1006 that indicate instances of previously performed surgicalprocedures. Each different instance of a first data informationstructures 1006 corresponds to a different surgery. Second datainformation structures 1008, associated with individual surgical events,associate motion picture image segments (MPI₁, MPI₂, . . . MPI_(n)) withkinematic information segments (KIN₁, KIN₂, . . . KIN_(n)), withsurgical system actuation states (Astate₁, Astate₂, . . . Astate_(n))and with corresponding annotations (ANOT₁, ANOT₂, . . . ANOT_(n)) atcorresponding time intervals (t₁, t₂, . . . t_(n)). First control signalrules information structure 1010, produced based upon a plurality ofsurgical events of a given type, associates kinematic/imagesignature/actuation state combinations with control signals e. g.,(SigK₁, SigI₁, Astate₁, CNTL_(KIA1)) . . . (SigK_(n), SigI_(n),Astate_(n), CNTL_(KIAn)). Each different instance of the first rulesinformation structure 1010 corresponds to a different category ofsurgery. Third data information structures 1012, associated withindividual surgical events, associate motion picture image segments(MPI₁, MPI₂, . . . MPI_(n)) with haptics information segments (HAP₁,HAP₂, . . . HAP_(n)) at corresponding time intervals (t₁, t₂, . . .t_(n)). Second diagnosis rules information structures 1014, producedbased upon a plurality of surgical events of a given type, associatediagnosis image signatures (SigI_(DIAG1), SigI_(DIAG2), . . .SigI_(DIAGm)) with diagnostic kinematic signatures (SigK_(DIAG1),SigK_(DIAG2), . . . SigK_(DIAGm)) and with diagnoses (DIAG₁, DIAG₂, . .. DIAG_(m)). Each different instance of the second diagnosis rulesinformation structure 1014 corresponds to a different category ofsurgery.

In accordance with some embodiments, image information can be in theform of video images across the full visual spectrum, fluorescence,hyperspectral, CT/MRI. Image information or a combination of some ofthese image information can be used as a basis to evaluate diseasestate/determine diagnosis.

FIG. 11 is an illustrative drawing representing an example instance of afirst data information structure 1006 included within the atlas 1002 inthe storage device 1004, which includes information about an individualsurgical procedure in accordance with some embodiments. It will beappreciated that a multiplicity of surgical procedures are performed ona multiplicity of different patients by many different surgeons usingmany different instances of a robot-assisted surgical system describedherein. An instance of the first data information structure 1006 can beproduced and stored in a computer readable storage device for eachsurgical procedure that is performed. The example first data informationstructure 1006 includes a surgery type field 1006-1 that indicates thetype of surgical procedure, such as prostectomy, hysterectomy, or,partial-nephrectomy, for example.

The example first data information structure 1006 includes a patienthealth record field 1006-2 that provides information about the patientwho is operated upon such as age, body mass, blood type, height, sex,and race, for example. The example first data information structure 1006includes a physician information field 1006-3 that provides informationabout the surgeon performs the individual operation such as level ofexperience in general and level of experience operating a robot-assistedsurgical system, for example. The example first data informationstructure 1006 includes a surgical system field 1006-4 that providesinformation about the surgical system used to perform the operation suchas make, model and serial number, for example. The example first datainformation structure 1006 includes a surgical recording field 1006-5that provides information such as motion picture images, instrumentkinematics and haptic feedback provided using the system during thesurgical procedure. The example first data information structure 1006includes a surgical recording field 1006-6 that provides annotationinformation, such as tags containing descriptive information that havebeen associated with image information, kinetics information or hapticsinformation within the example first data information structure 1006.

FIG. 12 is an illustrative drawing representing an example instance ofthe second data information structure 1008 included within the atlas1002 in the storage device 1004, which associates recorded motionpicture image segments from an individual surgical procedure,corresponding surgical instrument kinematics information segments,corresponding surgical system actuation states, and correspondingannotations, in accordance with some embodiments. The example seconddata information structure 1008 provides additional details ofinformation within the recording field 1006-5 of the first datainformation structure 1006. In one aspect, motion picture images ofpatient anatomy structures, corresponding surgical instrumentkinematics, and corresponding surgical system actuation states, arerecorded and time stamped (t1, t2 . . . tn) during a surgery to producea chronological record of surgical activities and corresponding surgicalinstrument kinematics during the surgical procedure. In someembodiments, the motion picture images also encompass images of surgicalinstruments used to operate upon the anatomical structures during asurgery. Time stamps temporally align motion picture images withsurgical instrument kinematics.

During a surgery, a user such as a surgeon or another member of asurgical team, may annotate recorded motion picture information andassociated recorded surgical instrument kinematics information withmetadata that indicate corresponding surgical activity such asdissection or suturing, anatomical features, particular surgicalcomplexities, surgeon observations, or phase of the procedure, forexample. The annotations may include one or more of or a combination ofwritten notes tagged to recorded motion picture information and/orrecorded surgical instrument kinematics information, coloring orhighlighting (e.g., telestration) of images in the video recordings, forexample. The annotations may be added later to a time stamped recordedimage information and/or recorded kinematic information, for example.

It will be appreciated that during a teleoperated surgical procedure, asurgical activity can occur that results in a change in surgical systemactuation state. A surgeon may move his head into and out of the viewer31 resulting in a change in head-in state. A surgeon may move his handsor feet in and out of contact with control inputs 36 resulting in achange in following state, for example. A combination of instruments inuse may be changed, resulting in a change in instruments type state, forexample.

Operation of the surgical instrument in support of the surgical activityin the one or more surgical states results in generation of kinematicinformation within a surgical system that indicates instrument motion,which is indicative of the actual manner in which a surgeon performedthe surgical activity. A surgeon may have moved an instrument rapidly orslowly, for example. A surgeon may have moved a surgical instrument in adirection toward or in a direction away from an anatomical structurealong one or another path, for example. A surgeon, before actuating aparticular instrument, may have adjusted a position of a differentinstrument, for example. It will be appreciated that a combination ofrecorded anatomical structure image information and recorded instrumentkinematic information, and recorded system actuation state provides arecord of what images were presented to a surgeon during a surgery andwhat activities the surgeon engaged in in concert with those images andwhat the system actuation state of the surgical system was at the time.Corresponding annotations can provide additional insight into associatedimages and kinematics recorded in the course of a surgical procedure.Note that while kinematic information can be derived directly from thesurgical system (e.g., via joint data or other mechanical tracking), invarious other embodiments, kinematic information can be extracted fromimage processing or sensor data (e.g., tool/instrument tracking withinthe endoscope image).

FIGS. 13A-13C are illustrative drawings showing an example surgicalinstrument 1202 and an actuator assembly 1203 in which the surgicalinstrument is shown in three different example surgical instrumentactuation states in accordance with some embodiments. The exampleinstrument 1202 includes a jaw end effector 1204 that can transitionbetween open and closed states and a continuum of partiallyopened/partially closed states in between. The example instrument 1202also includes a two degree of freedom (2-dof) wrist 1206 that can movebetween different two-dimensional (x, y) positional states. The exampleactuator assembly 1203 includes a first actuator 1208, which in someembodiments includes a jaw motor (JM) used to actuate the jaw endeffector 1204. The example actuator assembly 1203 includes a secondactuator 1210, which in some embodiments includes a wrist motor (WM)used to actuate the wrist 1206. During a surgery, the surgicalinstrument 1202 may transition through multiple instrument actuationstates corresponding to different activities during a surgicalprocedure. Each transition results in generation of kinematicsinformation that is captured and stored and that is indicative of motionof the instrument as it transitions from its physical location anddisposition (e.g., open or closed) in one state to its physical locationand disposition in a next state. As represented in FIG. 13A, forexample, a surgical procedure may involve a first surgical activity inwhich the first actuator 1208 (the JM) disposes the jaw end effector1204 to a fully open state and the second actuator 1210 the (WM)disposes the wrist 1206 to a first positional state (x1, y1). Asrepresented in FIG. 13B, for example, the surgical procedure may involvea second surgical activity in which the first actuator 1208 transitionsthe jaw end effector 1204 to a fully closed state and the secondactuator 1210 transitions the wrist 1206 to a second positional state(x2, y2). As represented in FIG. 13C, for example, the surgicalprocedure may involve a third surgical activity in which the firstactuator 1208 disposes the jaw end effector 1104 in a partiallyopen/partially closed state and the second actuator 1210 transitions thewrist 1206 to a third positional state (x3, y3).

FIG. 14 is an illustrative drawing representing an example instance ofthe first control signal rules information structure 1010 includedwithin the atlas 1002 in the storage device 1004, which includes rulesto associate kinematic signature information, image signatureinformation, and system actuation state information with controlsignals, in accordance with some embodiments. The rules are developedbased upon data from prior surgeries represented in the first and seconddata information structures 1006, 1008. The first rules correlatepatterns of anatomical images, instrument kinematics and system statewith control signals used to control operation of the system or toprovide guidance to a surgeon during a surgical procedure.

In accordance with some embodiments, a kinematic signature includes amulti-dimensional vector. In some embodiments, recorded kinematic motionof an instrument is decomposed into multiple vectors representingkinematic features such as instantaneous velocity, instantaneousacceleration, instantaneous three-dimensional positon, current path ofmotion and predicted path of motion, for example. Not only can themotion and position of an instrument, but also its context such asphysical location of an anatomical structure relative to the instrument,physical location of other instruments, a patient's health and thenature of a surgery be relevant to interpretation of an kinematicinformation. Moreover, previous instrument motions can be relevant to anevaluation, such as where to move an instrument next, that is to bebased at least in part upon instrument kinematics. Thus, in someembodiments, an instrument kinematics vector also includes vectorsindicative of location of anatomical structures, location of otherinstruments, patient health, type of surgery and prior motion of aninstrument, for example.

In accordance with some embodiments, an image signature includes amulti-dimensional vector. In some embodiments, recorded motion pictureimages of an anatomical structure are decomposed into multiple vectorsrepresenting image features such as color, texture and geometry, forexample. Moreover, in a convolutional neural network, there arelower-level features (e.g., color edges, etc.); in subsequent layers,higher level features are learned, ultimately resulting in e.g., aclassification of the anatomical structure or tissue type. Not only canthe appearance of an anatomical structure, but also its context such aspatient's health, the nature of a surgery, a surgeon's skill level atthe particular type of robot assisted surgery, and whether or not systemstate indicates that the correct surgical instruments currently areinstalled in the system, be relevant to interpretation of an anatomicalimage. Moreover, changes in appearance of an anatomical structure in thecourse of a surgical procedure can be relevant to interpretation of ananatomical image. Thus, in some embodiments, an image vector alsoincludes vectors indicative of patient health record information,surgery type, and comparison of anatomical image appearance at differentsurgical stages, for example.

In accordance with some embodiments, different control signals areassociated with different combinations of image signatures, kinematicssignatures and actuation system state. In some embodiments, some controlsignals control instrument actuation state. For example, somecombination of image signature, kinematics signature and actuationsystem state may correspond to an unsafe surgical activity, and acorresponding control signal may operate to cause a surgical system tofreeze motion of an instrument and/or generate a notification to asurgeon (e.g., a visible, audible, and/or tactile warning , or anindication that related guidance/information is available). For example,instrument motion outside the field of view of the camera.Alternatively, for example, some combination of image signature,kinematics signature and actuation system state may correspond to anespecially delicate surgical activity, and a corresponding controlsignal may operate to cause a surgical system to slow rate of motion ofan instrument to a safer speed or may limit a range of motion of aninstrument to avoid injury or may generate a recommendation notificationto a surgeon (e.g., overlay of corrective/warning information displayedwithin a viewer or presentation of access to relatedguidance/information). For example, preservation of nerves duringprostatectomy, or suturing during mitral valve repair.

In accordance with some embodiments, machine learning techniques can beused to generate image signatures and to generate kinematics signatures.More specifically, for example, classifiers can be used together withexpert knowledge to correlate image signatures and kinematics signatureswith control signals. Surgical data within the first and second datainformation structures 1006, 1008 are evaluated, based upon expertsurgeon input for example, to determine appropriate system operation incontext of different anatomical images, instrument kinetics and surgicalsystem state. Control signals to effect the determined system operationare associated in the first rules information structure 1010 withanatomical image signatures, instrument kinematics signatures and systemstates. In accordance with some embodiments, image signatures andkinematics signatures can be combined together with system stateinformation to produce one overall signature that corresponds to acontrol signal.

FIG. 15 is an illustrative flow diagram representing a process 1502 toproduce a control signal based at least in part upon anatomical imageinformation, instrument kinematics and system state, in accordance withsome embodiments. The computer processor 58 is configured to perform theprocess 1502 in accordance with some embodiments. During performance ofa surgical procedure, a rules block 1504 determines whether to cause alaunch of a control signal based upon motion picture images, instrumentkinematics and surgical system state system state during the surgery,and rules from the first rules information structure 1010.

More particularly, during performance of a surgical procedure usingsystem 10, rules block 1504 receives motion picture image informationgenerated during the surgery, receives instrument kinematics informationgenerated during the surgery and receives system state informationgenerated during the surgery. The camera 528 captures motion pictureinformation during the surgery. The computer processor 58 at block 1506,provides corresponding image information to the rules block 1504.Instrument actuators, such as actuators 1208, 1210, receive actuationcommands that result in instrument motion during a surgery. The computerprocessor 58 at block 1508, determines instrument kinematic informationbased upon the actuation commands and provides the kinematic informationto the rules block 1504. The system includes sensors (not shown) thatsense system state, such as whether an operator has his head placedagainst the viewer, whether the operator has engaged control inputs withhands and/or feet 36, and which instruments are inserted for use. Thecomputer processor 58 at block 1510, provides the system stateinformation to the rules block 1504.

Also, during the surgical procedure, a control signal rules storageblock 1512 provides to the rules block 1504, image signatureinformation, kinematics signature information, and system actuationstate information from within a first rules block portion 1010A of thefirst control signal rules information structure 1010. The rules block1504 compares image, kinematics and system information provided byblocks 1506, 1508, 1510, respectively, with associated image signature(SigI), kinematics signature (SigK) and system state (Astate)information from the first rules block portion 1010A. It will beappreciated that in some embodiments, the computer processor 58 isconfigured to transform image, kinematics and system informationgenerated by the system 10 into a format suitable for comparing againstsignature and state information from the first portion 1010A of thecontrol signal rules information structure 1010. In particular, in someembodiments raw image/kinematics/system state is processed to derive aclassification (signal/probability) which then is looked up in a tableto determine what action to take based upon a determined classification.

Decision module 1506 determines whether a match occurs between theprovided image, kinematics and system state information and rulesinformation from the first rules block portion 1010A. It will beappreciated that in machine learning embodiments a match is determinedbased upon a range of similarity between the image, kinematics andsystem state information and rules information. Thus, for example, acombination of generated image, generated kinematics and generatedsystem state information that is within some threshold limit of acertain rule is determined to match that rule.

Block 1514, in response to determination of a match between acombination of image, kinematics and system state information and arule, launches a control signal from within a second rules portion 1010Bof the control signal rules information structure 1010 that correspondsthe matching rule. For example, in response to determination that image,kinematics and system state information received during a surgicalprocedure matches SigI₂, SigK₂ and Astate₂ of the first portion 1010A,block 1516 launches signal CNTL_(IKA2) from within a second portion 101Bof the control signal information structure 1010. Thus, the launchedcontrol signal depends upon actual system information. Moreover, alaunched control signal can be operative to control instrumentactuation. A control signal can be launched that prevents certaininstrument motion, such as to avoid collision with another instrument,for example. A control signal can be launched that controls how certaininstrument motion occurs, such as by limiting rate of speed, forexample. A control signal can be launched that controls what aninstrument does next such as moving an instrument to a neutral positionto prevent a collision with an instrument that a surgeon is likely touse next, for example. While no match is detected, decision module feedsback control flow to the rules block 1504. Similarly, after the launchof a control signal, control flows back to the rules block 1504 tocontinue with comparisons of image, kinematics and system informationgenerated during a surgery with rules from the control signalinformation structure 1010.

FIG. 16 is an illustrative drawing representing an example instance ofthe third data information structure 1012 included within the atlas 1002in the storage device 1004, which associates recorded motion pictureimage segments with haptics information segments, in accordance withsome embodiments. The example third data information structure 1012provides additional details of information within the recording field1006-5 of the first data information structure 1006. In one aspect,motion picture images of patient anatomy structures and correspondinghaptic feedback information imparted to a surgeon/operator are recordedand time stamped (t1, t2 . . . tn) during a surgery to produce achronological record of what a surgeon observed and what a surgeonsensed through a physical touch sensation at the control input duringthe surgical procedure. Time stamps temporally align motion pictureimages with surgical instrument kinetics.

Haptics generally describes touch feedback, which may includekinesthetic (force) and cutaneous (tactile) feedback, as well asvibration and movement. In teleoperation surgery systems, natural hapticfeedback is largely eliminated because a surgeon does not manipulate aninstrument directly. Referring to FIG. 6 , in accordance someembodiments, an artificial haptic sensor 660 mounted on a patient-sideof a surgical instrument 520 can acquire haptic information, and hapticdisplays on the surgeon side to convey the sensed information to thesurgeon. Referring to FIG. 5A, Kinesthetic or force feedback systemstypically measure or estimate the forces applied to patient anatomywithin a surgical scene by the surgical instrument, and provide resolvedforces to a surgeon's hand via a force feedback device 39 mounted on acontrol input 36. Tactile feedback can be used to provide informationsuch as local tissue deformation and pressure the distribution across atissue surface.

Referring again to FIG. 16 , third data information structure 1012associates motion picture information segments with contemporaneouslyhaptic feedback information delivered to force feedback device 39. Assuch, the data in the third information structure can represent a recordof a surgeon's visual and tactile input during a surgical procedure.This information can be used to aid in training a surgeon in mechanicaltechnique and also as an aid in training a surgeon in diagnosis.

FIG. 17 is an illustrative flow diagram representing a process 1702 toconfigure a teleoperated robot-assisted surgical system to playback asurgical experience, in accordance with some embodiments. The computerprocessor 58 is configured to perform the process 1702 in accordancewith some embodiments. The processor 58 uses the recorded motion pictureinformation to configure the camera 528 to replay the recorded motionpictures at the viewer. The processor 58 uses the recorded hapticinformation to configure the force feedback device 39 to impart therecorded haptic feedback at the control input 36. In some embodiments,the force feedback device 39 includes a vibrotactile device.

It will be appreciated that such replay can be helpful in training asurgeon to more effectively use a surgical system based upon thatsurgeon's own prior surgical experience or based upon that of another.By looking into the viewer 31 and touching the control input 36 whilethe images and corresponding haptic feedback forces are replayed, asurgeon can experience a recorded surgery. A surgeon then can experimentwith alternate surgical approaches and compare the visual and tactilesensations during those alternate approaches with those of a priorsurgery, for example. FIG. 18 is an illustrative drawing representing anexample instance of the second diagnosis rules information structure1014 included within the atlas 1002 in the storage device 1004, whichincludes rules to associate image signature information and hapticfeedback information with diagnoses, in accordance with someembodiments. The rules are developed based upon data from priorsurgeries represented in the first and third data information structures1006, 1012. The second rules correlate patterns of anatomical images andhaptic feedback with diagnoses.

As explained above, an image signature includes a multi-dimensionalvector. Additionally, in accordance with some embodiments, a hapticfeedback signature includes a multi-dimensional vector. In someembodiments, recorded haptic feedback is decomposed into multiplevectors representing haptic feedback features such as location,frequency and intensity of feedback forces. Not only can the location,frequency and intensity of feedback forces, but also its context suchas, a patient's health and the nature of a surgery be relevant tointerpretation of a feedback signal. Thus, in some embodiments, amulti-dimensional haptic feedback vector also includes vectorsindicative of patient health and type of surgery, for example.

In accordance with some embodiments, machine learning techniques can beused to generate image signatures and to generate haptic feedbacksignatures. More specifically, for example, classifiers can be usedtogether with expert knowledge to correlate image signatures and hapticsignatures with diagnoses. Surgical data within the first and third datainformation structures 1006, 1012 are evaluated, based upon expertsurgeon input for example, to determine appropriate diagnoses in contextof different anatomical images, and haptic feedback forces. Inaccordance with some embodiments, image signatures and hapticssignatures can be combined together to produce one overall signaturethat corresponds to a diagnosis.

FIG. 19 is an illustrative flow diagram representing a process 1902 toproduce a diagnosis based at least in part upon anatomical imageinformation and haptic feedback information, in accordance with someembodiments. The computer processor 58 is configured to perform theprocess 1902 in accordance with some embodiments. During performance ofa surgical procedure, a rules block 1904 determines whether to report adiagnosis based upon motion picture images and haptic feedback forcesduring the surgery, and rules from the second rules informationstructure 1014.

More particularly, during performance of a surgical procedure usingsystem 10, rules block 1904 receives motion picture image informationgenerated during the surgery and receives haptic feedback informationgenerated during the surgery. The camera 528 captures motion pictureinformation during the surgery. The computer processor 58 at block 1906,provides corresponding image information to the rules block 1904. Thehaptic feedback force device 39 generates haptic feedback force. Thecomputer processor 58 at block 1908, determines haptic feedback forceinformation based upon the actuation commands and provides the kinematicinformation to the rules block 1904.

Also, during the surgical procedure, a diagnosis rules storage block1912 provides to the rules block 1904 image signature information andhaptic feedback information from a first rules block portion 1014A ofthe diagnosis rules information structure 1014. The rules block 1904compares image and haptic feedback information provided by blocks 1906,1908, respectively, with associated image signatures (SigI) and hapticsfeedback signatures (SigH) from the first rules block portion 1014A ofthe rules diagnosis structure 1014. It will be appreciated that in someembodiments, the computer processor 58 is configured to transform imageand haptic feedback information generated by the system 10 into a formatsuitable for comparing against signature and state information from thefirst portion 1014A of the diagnosis rules information structure 1014.

Decision module 1914 determines whether a match occurs between theprovided image and haptic feedback information and rules informationfrom the first rules portion 1014A of the diagnosis rules informationstructure 1014. It will be appreciated that in machine learningembodiments a match is determined based upon a range of similaritybetween the image and haptic information and rules information. Thus,for example, a combination of generated image and generated hapticsinformation that is within some threshold limit of a certain rule isdetermined to match that rule.

Block 1914, in response to determination of a match between acombination of image, kinematics and system state information and arule, launches a control signal from within a second rules block 1010Bportion of the first control signal rules information structure 1010that corresponds the matching rule. For example, in response todetermination that image and haptic feedback information received duringa surgical procedure matches SigI₂ and SigH₂ of a first portion 1014A ofthe diagnosis information structure 1014, block 1916 reports on theviewer 31, diagnosis DIAG1 form a second portion of the diagnosisinformation structure 1014. Thus, a diagnosis during a surgery dependsupon actual system information generated during a surgery andinformation recorded in many prior surgeries. During the process 1902,while no match is detected, decision module 1914 feeds back control flowto the rules block 1904. Similarly, after a diagnosis, flow controlsback to the rules block 1904 to continue with comparisons of image andhaptics generated during a surgery with rules from the diagnosis rulesinformation structure 1014.

Although illustrative embodiments have been shown and described, a widerange of modification, change and substitution is contemplated in theforegoing disclosure and in some instances, some features of theembodiments may be employed without a corresponding use of otherfeatures. For example, in some embodiments, the processor 58 is coupledto a memory device such as storage device 1004 that includesinstructions to generate a virtual surgical system that includes avirtual surgical instrument and a virtual surgical instrument actuator.The memory device 1004 includes an instruction set executable on theprocessor 58 to cause the processor 58 to perform virtual operations. Insome embodiments, the virtual operations include receiving user inputcommands from a user to control movement of a first virtual roboticsurgical instrument. The virtual operations further include trackingvirtual surgical instrument actuator state of the first virtual roboticinstrument during movement of the first virtual robotic surgicalinstrument in response to the user input commands during a virtualsurgical procedure. The virtual operations include recording virtualsurgical instrument kinematic information indicative of virtual surgicalinstrument motion produced within the virtual surgical system during theoccurrence of the virtual surgical procedure. The virtual operationsinclude determining respective kinematic signatures associated withrespective virtual surgical instrument motions and producing aninformation structure in a computer readable storage device thatassociates respective kinematic signatures with respective controlsignals. The virtual operations further include comparing, during aperformance of the virtual surgical procedure within the virtualsurgical system, virtual surgical instrument kinematic informationduring the performance with at least one respective kinematic signature.The virtual operations further include launching, during a performanceof the virtual surgical procedure within the virtual surgical system, anassociated respective control signal within the virtual surgical systemin response to a match between virtual surgical instrument kinematicsduring the performance and a respective kinematic signature.

In yet another embodiment, a system produces a virtual surgical systemthat includes an information structure in a computer readable storagedevice that associates surgical image signatures with control signals. Aprocessor is configured to generate a virtual surgical system. Theprocessor is configured to compare, during a performance of a virtualsurgical procedure within the virtual surgical system, surgical imageswithin a surgical scene with at least one surgical image signature. Theprocessor is configured to launch, during the performance of the virtualsurgical procedure a control signal within in response to a matchbetween surgical images during the performance and the at least onesurgical image signature. The processor is configured to generate avirtual instrument configured to adjust its motion in response to thecontrol signal.

EXAMPLES

Example 1 includes a method for use with a virtual surgical system, themethod comprising: for a multiplicity of occurrences of a virtualsurgical procedure, recording virtual surgical instrument kinematicinformation indicative of virtual surgical instrument motion producedwithin the virtual surgical system during the occurrence of the virtualsurgical procedure; determining respective kinematic signaturesassociated with respective virtual surgical instrument motions;producing an information structure in a computer readable storage devicethat associates respective kinematic signatures with respective controlsignals; comparing, during a performance of the virtual surgicalprocedure within the virtual surgical system, virtual surgicalinstrument kinematic information during the performance with at leastone respective kinematic signature; launching, during a performance ofthe virtual surgical procedure within the virtual surgical system, anassociated respective control signal within the virtual surgical systemin response to a match between virtual surgical instrument kinematicsduring the performance and a respective kinematic signature.

Example 2 includes the method of claim Example 1 further including: inresponse to the control signal, adjusting speed at which a virtualinstrument moves.

Example 3 includes the method of Example 1 further including: inresponse to the control signal, adjusting a range of motion of a virtualinstrument.

Example 4 includes the method of Example 1 further including: inresponse to the control signal, providing a visual image providingguidance to an operator.

Example 5 includes a method for use with a virtual surgical system, themethod comprising: for a multiplicity of occurrences of a virtualsurgical procedure within a virtual surgical system, recording images ofa surgical scene within the virtual surgical system during theoccurrence of the virtual surgical procedure; determining respectivesurgical image signatures associated with respective images of surgicalscenes; producing an information structure in a computer readablestorage device that associates respective surgical image signatures withrespective control signals; comparing, during a performance of thevirtual surgical procedure within the virtual surgical system, surgicalimages within a surgical scene within the virtual surgical system withat least one respective surgical image signature; launching, during aperformance of the virtual surgical procedure within the virtualsurgical system, an associated respective control signal within thevirtual surgical system in response to a match between surgical imagesduring the performance and a respective surgical image signature.

Example 6 includes the method of Example 5 further including: inresponse to the control signal, adjusting speed at which a virtualinstrument moves.

Example 7 includes the method of Example 5 further including: inresponse to the control signal, adjusting a range of motion of a virtualinstrument.

Example 8 includes the method of Example 5 further including: inresponse to the control signal, providing a visual image providingguidance to an operator.

Example 9 includes a system to produce a virtual surgical systemcomprising: an information structure in a computer readable storagedevice that associates respective kinematic signatures with respectivecontrol signals; a processor configured to, generate a virtual surgicalsystem; compare, during a performance of a virtual surgical procedure,virtual instrument kinematic information generated during the virtualsurgical procedure with at least one surgical image signature; andlaunch, during the performance of the virtual surgical procedure acontrol signal in response to a match between virtual instrumentkinematic information generated during the virtual surgical procedureand the at least one surgical image signature; and generate a virtualinstrument configured to adjust its motion in response to the controlsignal.

Example 10 includes the system of Example 9, wherein in response to thecontrol signal, the virtual instrument is configured to adjust speed atwhich it moves.

Example 11 includes the system of Example 9, wherein in response to thecontrol signal, the virtual instrument is configured to adjust a rangeof motion of the instrument.

Example 12 includes the system of Example 9, wherein in response to thecontrol signal, the virtual instrument is configured to adjust visualguidance provided to an operator.

Example 13 includes a system to produce a virtual surgical systemcomprising: an information structure in a computer readable storagedevice that associates respective surgical image signatures withrespective control signals; a processor configured to, generate avirtual surgical system; compare, during a performance of a virtualsurgical procedure within the virtual surgical system, surgical imageswithin a surgical scene with at least one surgical image signature;launch, during the performance of the virtual surgical procedure acontrol signal within in response to a match between surgical imagesduring the performance and the at least one surgical image signature;and generate a virtual instrument configured to adjust its motion inresponse to the control signal.

Example 14 includes the system of Example 13, wherein in response to thecontrol signal, the virtual instrument is configured to adjust speed atwhich it moves.

Example 15 includes the system of Example 13, wherein in response to thecontrol signal, the virtual instrument is configured to adjust a rangeof motion of the instrument.

Example 16 includes the system of Example 13, wherein in response to thecontrol signal, the virtual instrument is configured to adjust visualguidance provided to an operator.

Example 17 includes a method for use with a telepoperated surgicalsystem (surgical system), the method comprising: for a multiplicity ofoccurrences of a surgical procedure within one or more instances of thesurgical system, recording images of a surgical scene within thesurgical system during the occurrence of the surgical procedure;determining respective surgical image signatures and kinematicsignatures associated with respective images of surgical scenes;producing an information structure in a computer readable storage devicethat associates respective surgical image signatures and kinematicsignatures with respective control signals; comparing, during aperformance of the surgical procedure within an instance of the surgicalsystem, surgical images within a surgical scene within the instance ofthe surgical system with at least one respective surgical imagesignature; comparing, during the performance of the surgical procedurewithin an instance of the surgical system, surgical instrument kinematicinformation during the performance with at least one respectivekinematic signature; launching, during the performance of the surgicalprocedure within the instance of the surgical system, an associatedrespective control signal within the surgical system in response to amatch between surgical images during the performance and a respectivesurgical image signature and a match between the surgical instrumentkinematics during the performance and a respective kinematic signature.

Example 18 includes a teleoperated surgical system comprising: aninformation structure in a computer readable storage device respectivesurgical image signatures and kinematic signatures with respectivecontrol signals; a processor configured to, compare, during aperformance of a surgical procedure, instrument kinematic informationgenerated during the surgical procedure with at least one kinematicsignature; compare, during a performance of a surgical procedure withinthe surgical system, surgical images within a surgical scene with atleast one surgical image signature; and launch a control signal inresponse to a match between the instrument kinematic informationgenerated during the surgical procedure and the at least one kinematicsignature and a match between surgical images within a surgical sceneand the at least one surgical image signature.

Example 19 includes the teleoperated surgical system of Example 18,further comprising an instrument configured to adjust its motion inresponse to the control signal.

1. (canceled)
 2. A surgical system comprising: one or more processors,in communication with an actuation system the surgical system, theactuation system configured to control a surgical instrument, thesurgical system configured to track kinematic information indicative ofmotion of the surgical instrument during a surgical procedure; whereinthe one or more processors are configured to: identify one or more imagesignatures for one or more images of a surgical scene during thesurgical procedure and one or more kinetic signatures of the surgicalinstrument based at least on the kinematic information of the surgicalinstrument tracked during the surgical scene; determine an instrumentstate; and selectively generate a notification provided to an operatorof the surgical system based at least on the one or more imagesignatures, the one or more kinematic signatures, and the instrumentstate.
 3. The system of claim 2, wherein the notification comprises oneof a visible, tactile or audible warning.
 4. The system of claim 2,wherein the one or more processors are further configured to generatethe notification as a recommendation notification to the operator, therecommendation notification comprising an overlay of informationdisplaying guidance information.
 5. The system of claim 2, wherein theone or more processors are further configured to cause, responsive tothe determination, the actuator system to slow a rate of the motion ofthe surgical instrument.
 6. The system of claim 2, wherein the one ormore processors are further configured to cause, responsive to thedetermination, the surgical system to freeze the motion of the surgicalinstrument.
 7. The system of claim 2, wherein the one or more imagesignatures comprise one or more vectors representing one or moredimensions of one or more images recorded for previous occurrences ofthe surgical procedure.
 8. The system of claim 2, wherein the one ormore kinematic signatures comprise one or more vectors representing oneor more dimensions of kinematic information of the surgical instrumentrecorded for previous occurrences of the surgical procedure.
 9. Thesystem of claim 2, wherein the one or more processors are furtherconfigured to identify the one or more image signatures based at leaston a comparison of the one or more images to the one or more recordedimage signatures of previous occurrences of the surgical procedure. 10.The system of claim 2, wherein the one or more processors are furtherconfigured to identify the one or more kinematic signatures based atleast on a comparison of the kinematic information to the one or morerecorded kinematic signatures of previous occurrences of the surgicalprocedure.
 11. The system of claim 2, wherein the one or more processorsare further configured to determine to selectively generate thenotification based at least on a rule, the rule associating the one ormore image signatures, the one or more kinematic signatures and thestate of the actuator system with a signal, the signal causinggeneration of the notification.
 12. A computer-implemented method,comprising: receiving, by one or more processors from a surgical system,kinematic information indicative of motion of a surgical instrumentduring a surgical procedure; receiving, by the one or more processors,one or more images of a surgical scene during the surgical procedure;identifying, by the one or more processors, one or more image signaturesfor the one or more images and one or more kinetic signatures of thesurgical instrument based at least on the kinematic information;determining, by the one or more processors based at least on the one ormore image signatures and the one or more kinematic signatures, anotification to generate; and generating, by the one or more processors,the notification to be provided to a surgeon associated with thesurgical procedure.
 13. The computer-implemented method of claim 12,wherein the notification comprises one of a visible, tactile or audiblewarning.
 14. The computer-implemented method of claim 12, furthercomprising generating, by the one or more processors, the notificationas a recommendation notification to the operator, the recommendationnotification comprising an overlay of information displaying guidanceinformation.
 15. The computer-implemented method of claim 12, furthercomprising causing, by the one or more processors, an actuator system ofthe surgical system to slow a rate of the motion of the surgicalinstrument.
 16. The computer-implemented method of claim 12, furthercomprising causing, by the one or more processors, the surgical systemto freeze the motion of the surgical instrument.
 17. Thecomputer-implemented method of claim 12, wherein the one or more imagesignatures comprise one or more vectors representing one or moredimensions of one or more images recorded for previous occurrences ofthe surgical procedure.
 18. The computer-implemented method of claim 12,wherein the one or more kinematic signatures comprise one or morevectors representing one or more dimensions of kinematic information ofthe surgical instrument recorded for previous occurrences of thesurgical procedure.
 19. The computer-implemented method of claim 12,further comprising generating, by the one or more processors, thenotification based at least on a rule, the rule associating the one ormore image signatures and the one or more kinematic signatures.
 20. Anon-transitory computer-readable medium comprising instructionsconfigured to cause a computer system to perform operations comprising:receiving from one or more sensors of a surgical system, kinematicinformation of a surgical instrument and one or more images of asurgical scene during a surgical procedure; identifying one or moreimage signatures for the one or more images and one or more kineticsignatures of the surgical instrument based at least on the kinematicinformation; determining, based at least on the one or more imagesignatures and the one or more kinematic signatures, a notification tobe provided to an operator of the surgical system; and providing,responsive to the determination, the notification to the operator of thesurgical system.
 21. The non-transitory computer-readable medium ofclaim 20, further comprising instructions configured to cause thecomputer system to: Identify a state of an actuator system of thesurgical instrument; and determining, based at least on the one or moreimage signatures, the one or more kinematic signatures and the state ofthe actuator system , the notification to be provided to the operator.